Although the human temporal lobe has been documented to participate in semantic processing of both verbal and nonverbal stimuli, the exact neural basis underlying the common and unique processing of the two modalities is unclear. Semantic dementia (SD), a disease with a semantic-selective deficit due to predominant temporal lobe atrophy is an ideal lesion model to address this issue. However, many previous studies of SD used an impure patient sample or did not appropriately control for common components between tasks. To overcome these limitations, the present study aims to identify amodal semantic hubs and modality-specific regions in the temporal lobe by investigating behavioral performance on a verbal modality task (word associative matching) and a nonverbal modality task (picture associative matching) and neuroimaging data in 33 SD patients. We found that the left anterior fusiform gyrus was an amodal semantic hub whose gray matter volume correlated significantly with both modalities. We also observed two verbal modality-specific regions (the left posterior inferior temporal gyrus and the left middle superior temporal gyrus) and a nonverbal modality-specific region (the right lateral anterior middle temporal gyrus) whose gray matter volume correlated significantly with one modality when performance on the other modality was partialled out. The results remained significant when we excluded a wide range of potential confounding variables. Furthermore, to confirm the observed effects, we compared the performance of left- and right-hemispheric-predominant atrophic patients on the verbal and nonverbal tasks. The left-predominant patients showed more severe deficits in performance of the verbal task than the right-predominant patients, whereas the two groups of patients presented comparable deficits in the performance of the nonverbal task. These findings refined the structure of semantic network in the temporal lobe, deepening our understanding of the critical role of the temporal lobe in semantic processing. © 2019 Elsevier Ltd