Seizure prediction of epileptic preictal period through electroencephalogram (EEG) signals is important for clinical epilepsy diagnosis. However, recent deep learning-based methods commonly employ intra-subject training strategy and need sufficient data, which are laborious and time-consuming for a practical system and pose a great challenge for seizure predicting. Besides, multi-domain characterizations, including spatio-temporal-spectral dependencies in an epileptic brain are generally neglected or not considered simultaneously in current approaches, and this insufficiency commonly leads to suboptimal seizure prediction performance. To tackle the above issues, in this paper, we propose Contrastive Learning for Epileptic seizure Prediction (CLEP) using a Spatio-Temporal-Spectral Network (STS-Net). Specifically, the CLEP learns intrinsic epileptic EEG patterns across subjects by contrastive learning. The STS-Net extracts multi-scale temporal and spectral representations under different rhythms from raw EEG signals. Then, a novel triple attention layer (TAL) is employed to construct inter-dimensional interaction among multi-domain features. Moreover, a spatio dynamic graph convolution network (sdGCN) is proposed to dynamically model the spatial relationships between electrodes and aggregate spatial information. The proposed CLEP-STS-Net achieves a sensitivity of 96.7% and a false prediction rate of 0.072/h on the CHB-MIT scalp EEG database. We also validate the proposed method on clinical intracranial EEG (iEEG) database from our Xuanwu Hospital of Capital Medical University, and the predicting system yielded a sensitivity of 95%, a false prediction rate of 0.087/h. The experimental results outperform the state-of-the-art studies which validate the efficacy of our method. Our code is available at https://github.com/LianghuiGuo/CLEP-STS-Net.