Individuals with semantic dementia (SD) typically suffer from selective semantic deficits due to degenerative brain atrophy. Although some brain regions have been found to be correlated with the semantic impairments of SD patients, it is unclear if the damage is actually responsible for SD patients' semantic disorders because these findings were primarily obtained by examining the roles of local individual regions themselves without considering the influence of other regions that are functionally or structurally connected to the local individual regions. To resolve this problem, we investigated, from the brain network perspective, the relationship between the brain-network measures of regions and connections with semantic performance in 17 SD patients. We found that the severity of semantic deficits of SD patients was significantly correlated with the degree centrality values of the left anterior hippocampus (aHIP). Moreover, the semantic performance of the patients was also significantly correlated with the strength of gray matter functional connectivity of this region and two other regions: the left temporal pole/insula (TP/INS) and the left middle temporal gyrus. We further observed that the strength of the white matter structural connectivity of the left aHIP-left TP/INS tract could effectively predict the semantic performance of SD patients. When we controlled for a wide range of potential confounding factors (e.g., total gray matter volume), the above effects still held well. These findings revealed the critical brain network with the left aHIP as the center that could be contributing to the semantic impairments of SD.