Whole-brain segmentation from T1-weighted magnetic resonance imaging (MRI) is an essential prerequisite for brain structural analysis, e.g., locating morphometric changes for brain aging analysis. Traditional neuroimaging analysis pipelines are implemented based on registration methods, which involve time-consuming optimization steps. Recent related deep learning methods speed up the segmentation pipeline but are limited to distinguishing fuzzy boundaries, especially encountering the multi-grained whole-brain segmentation task, where there exists high variability in size and shape among various anatomical regions. In this article, we propose a deep learning-based network, termed Multi-branch Residual Fusion Network, for the whole brain segmentation, which is capable of segmenting the whole brain into 136 parcels in seconds, outperforming the existing state-of-the-art networks. To tackle the multi-grained regions, the multi-branch cross-attention module (MCAM) is proposed to relate and aggregate the dependencies among multi-grained contextual information. Moreover, we propose a residual error fusion module (REFM) to improve the network's representations fuzzy boundaries. Evaluations of two datasets demonstrate the reliability and generalization ability of our method for the whole brain segmentation, indicating that our method represents a rapid and efficient segmentation tool for neuroimage analysis.