Tests of verbal fluency have been widely used to assess the cognitive functioning of persons, and are typically classified into two categories (semantic and phonological fluency). While widely-distributed divergent and convergent brain regions have been found to be involved in semantic and phonological fluency, the anatomical connectivity underlying the fluency is not well understood. The present study aims to construct a comprehensive white-matter network associated with semantic and phonological fluency by investigating the relationship between the integrity of 22 major tracts in the whole brain and semantic fluency (measured by 3 cues) and phonological fluency (measured by 2 cues) in a group of 51 stroke patients. We found five left-lateralized tracts including the anterior thalamic radiation (ATR), inferior fronto-occipital fasciculus (IFOF), uncinate fasciculus (UF), superior longitudinal fasciculus (SLF) and frontal aslant tract (FAT) were significantly correlated with the scores of both semantic and phonological fluencies. These effects persisted even when we rifled out the influence of potential confounding factors (e.g., total lesion volume). Moreover, the damage to the first three tracts caused additional impairments in the semantic compared to the phonological fluency. These findings reveal the white matter neuroanatomical connectivity underlying semantic and phonological fluency, and deepen the understanding of the neural network of verbal fluency.