Spatio-temporal filtering has been widely used for extracting discriminative features in the motor imagery-based brain-computer interface (MI-BCI). In order to obtain high performance, the algorithms need to enhance robustness or find class-discriminative bands for the spatial filter. However, the existing methods either cannot derive the spatial and spectral filters with a unique objective function for guaranteeing convergence or rarely consider the combined optimization of spatial-spectral filters and other patterns for enhancing the discrimination. In this study, we present a novel feature extraction method termed Spectrum-weighted Tensor Discriminant Analysis (SwTDA), which optimizes spectral filters along with spatial filters and other associated patterns by tensor-based discriminant analysis. The proposed method considers intrinsic spatial-spectral-temporal information contained by the physiological signal and hence can identify discriminative characteristics robustly. The effectiveness of the algorithm is demonstrated by comparing it with several state-of-the-art methods on two datasets involving 15 different subjects. Results indicate that the SwTDA method yields higher classification accuracies than the competing methods. Furthermore, interpretable spatial-spectral patterns that are determined by the algorithm can be used for further analysis of the MI-based EEG signal.