Freezing of gait (FoG) is a common gait disorder in patients with the Parkinson's disease (PD), resulting in mobility limitation, risk of falls and decrease of daily life quality. Wearable sensors offer promising means of predicting FoG in living conditions and provide a critical time window for exogenous intervention. However, existing FoG prediction methods, suffering from complex hand-crafted feature design, low prediction rates, and high false positive rates, are far from ideal. In this study, FoG prediction with wearable inertial sensors is studied. A novel neural network model, named FoG-Net, is proposed to improve FoG prediction, avoiding complicated hand-crafted feature extraction. The FoG-Net consists of a backbone network and a feature fusion network, where the backbone network extracts shallow temporal features, and the feature fusion network automatically learns intra-token information using a self-attention mechanism. In order to comprehensive evaluate the performance of FoG prediction, a set of more practical metrics are designed, considering both prediction accuracy and risk of false alarm. Based on the new metrics, the FoG-Net achieves a competitive performance of 96.97% prediction rate and 22.73% false alarm rate on real FoG data, which provides a new potential for PD patients' fall prevention in daily life.