Great concerns have led to the evaluation of the potential hazards of nanosilica to human health and the environment. However, there still exists persistent debates on the biological effects and toxic consequences induced by nanosilica. The present study investigated both autophagy and apoptosis in ICR mice and Human hepatocellular carcinoma cells (HepG2), and then explored the interactive mechanism between these two distinct cell death modalities in HepG2 cells. Mice liver injuries seen by hematoxylin and eosin (HE) staining indicated the hepatotoxic effects of nanosilica. The TUNEL assay and immunohistochemistry results confirmed that nanosilica could induce both apoptosis and autophagy in vivo. Flow cytometry analysis demonstrated apoptosis induction in vitro, while autophagic ultrastructures, LC3-II expression and immunofluorescence clarified autophagy activation by nanosilica. Apoptosis suppression by the autophagy inhibitor of 3-methyladenine (3-MA) implied that autophagy was involved in apoptotic cell death. A mechanistic study verified that nanosilica induced autophagy via negative regulation of mammalian target of rapamycin (mTOR) signaling but not the Beclin-1 associated pathway. The enhancement of p62 accumulation and mTOR down-regulation might account for the molecular mechanism in contribution of autophagy to apoptosis. As an emerging new mechanism of nanomaterial toxicity, autophagy might be a more susceptive indicator for toxicological consequence evaluation in nanoparticle toxicity. The present study provides novel evidence to elucidate the toxicity mechanisms and may be beneficial to more rational applications of nanosilica in the future.