Objectives: Epileptic patients suffer from seizure recurrence after surgery due to the challenging localization. Improvement of the noninvasive imaging-based approach for a better definition of the abnormalities would be helpful for a better outcome. Methods: The quantitative anisotropy (QA) of diffusion spectrum imaging (DSI) is a quantitative scalar of evaluating the water diffusivity. Herein, we investigated the association between neuronal diameters or density acquired in literature and QA of DSI as well as the seizure localization in temporal lobe epilepsy. Thirty healthy controls (HCs) and 30 patients with hippocampal sclerosis (HS) were retrospectively analyzed. QA values were calculated and interactively compared between the areas with different neuronal diameter/density acquired from literature in the HCP-1021 template. Diagnostic tests were performed on Z-transformed asymmetry indices (AIs) of QA (which exclude physical asymmetry) among HS patients to evaluate its clinical value. Results: The QA values in HCs conformed with different pyramidal cell distributions ranged from giant to small; corresponding groups were the motor-sensory, associative, and limbic groups, respectively. Additionally, the QA value was correlated with the neuronal diameter/density in cortical layer IIIc (correlation coefficient with diameter: 0.529, p = 0.035; density: -0.678, p = 0.011). Decreases in cingulum hippocampal segments (Chs) were consistently observed on the sclerosed side in patients. The area under the curve of the Z-transformed AI in Chs to the lateralization of HS was 0.957 (sensitivity: 0.909, specificity: 0.895). Interpretation: QA based on DSI is likely to be useful to provide information to reflect the neuronal diameter/density and further facilitate localization of epileptic tissues.