Acupuncture therapy is an important branch of traditional Chinese medicine. Identifying manual acupuncture manipulation (MAM) poses significant challenges due to the variability in manual manipulations performed by different practitioners. These variations stem from differences in acupuncture training schools, individual experience, and personal interpretation. Even for the same type of MAM, there are numerous differences in the hand posture and the characteristics of the needle-holding fingers applying force. This complexity makes accurate MAM recognition difficult, particularly for assessing young practitioners' MAM techniques from multiple perspectives, including hand movement patterns and force application characteristics. To address these challenges, we developed a multi-source manual acupuncture manipulation multilayer recognition network (MS-MAM-MRN) to recognize MAMs using single-source data (tactile or visual) and fused multi-source data (tactile and visual). First, we designed a data augmentation and feature extraction method for single-source MAM data. We then developed a deep learning modeling framework for multilayer networks with multiple classifiers trained in parallel, and defined guiding labels for multi-label classification, and constructed a reconstruction method for training datasets indexed by these labels. Through dataset reconstruction, this study made the classification complexity faced by classifiers in multi-layer classification networks decrease with each layer, improving the accuracy of each classifier layer to over 90%. Lastly, we proposed a feature selection algorithm that combines hierarchical clustering and reinforcement learning to interpret the contribution of different features in MAM recognition. To validate our approach, we conducted experiments involving 50 senior acupuncturists from three institutions and 200 postgraduate and 300 undergraduate students. The collected tactile and visual signal data supported the evaluation of the MAM recognition model. Experimental results demonstrated that for four typical MAMs, our proposed model achieved a recognition accuracy of 95.3%, confirming its validity and effectiveness.