The intima-media thickness (IMT) of the carotid artery is commonly used for monitoring atherosclerosis. However, the intima-media complex (IMC) segmentation for the IMT calculation is a tedious task due to confused IMC boundaries and class-imbalance issues. In this paper, we propose an automatic method named CSM-Net for the joint segmentation of IMC on near and far walls, and Lumen in carotid ultrasound images. In the encoder-decoder CSM-Net, firstly, the cascaded dilated convolutions combined with the squeeze-excitation module are introduced for exploiting more contextual features on the last encoder layer. Secondly, a multi-scale triple spatial attention module is utilized for capturing serviceable features on each decoder layer. Lastly, a weighted hybrid loss function is employed to resolve the class-imbalance issue. Experiments are performed on a private dataset of 100 images from one center using the 10-fold cross-validation, the results of the proposed method on the IMC Dice, Lumen Dice, Precision, Recall, and F1 metrics are 0.814 +/- 0.061, 0.941 +/- 0.024, 0.911 +/- 0.044, 0.916 +/- 0.039, and 0.913 +/- 0.027, respectively, which precede some cutting-edge methods. The proposed method may be useful for the IMC segmentation of carotid ultrasound images in the clinic.