Accurate automatic sleep staging is crucial for diagnosing sleep disorders. However, most existing automatic data-driven sleep staging methods could not perfectly learn the complex knowledge of sleep staging criteria such as the American Academy of Sleep Medicine (AASM) based on the limited labeled data. This paper proposes a novel multimodal and multiscale automatic sleep staging framework, MMS-SleepNet, which explicitly incorporates AASM knowledge. It employs a deep learning multimodal feature extraction module (MMS-FE), embedding expert knowledge to effectively capture multimodal features for each stage and fine-grained EEG features at various frequencies. The module utilizes an attention mechanism to seamlessly fuse extracted multimodal features, significantly enhancing classification accuracy. To further improve the performance of MMS-SleepNet, a contrastive learning module and a data balancing strategy are proposed, addressing class confusion and data imbalance issues in existing models. Specifically, the contrastive classification module (CCM) emphasizes intra-class similarity and inter-class disparity, effectively alleviating class confusion. A simple yet effective data balancing mechanism augments the number of samples for the N1 sleep stage, guaranteeing that the model is trained on amore balanced dataset and proficiently resolves the long-tail distribution problem stemming from class imbalance. Experimental results on two public datasets validate the effectiveness of MMS-SleepNet, achieving a remarkable accuracy of 92.9% on the Sleep-EDF-20 dataset, surpassing other methods. Notably, it attains a 74.1% accuracy in the challenging N1 stage, outperforming other methods by 19.6-49.2%.