Motor Imagery (MI) is a typical paradigm for Brain-Computer Interface (BCI) system. In this paper, we propose a new framework by introducing a tensor-based feature representation of the data and also utilizing a convolutional neural network (CNN) architecture for performing classification of MI-EEG signal. The tensor-based representation that includes the structural information in multi-channel time-varying EEG spectrum is generated from tensor discriminant analysis (TDA), and CNN is designed and optimized accordingly for this representation. Compared with CSP+SVM (the conventional framework which is the most successful in MI-based BCI) in the applications to the BCI competition III-IVa dataset, the proposed framework has the following advantages: (1) the most discriminant patterns can be obtained by applying optimum selection of spatial-spectral-temporal subspace for each subject; (2) the corresponding CNN can take full advantage of tensor-based representation and identify discriminative characteristics robustly. The results demonstrate that our framework can further improve classification performance and has great potential for the practical application of BCI. © 2019 IEEE.