The generation of Traditional Chinese Medicine (TCM) prescription is one of the most challenging tasks in the research of intelligent TCM. Although there is a small part of research in this field, they usually use transfer learning methods to apply the relevant technology of text generation to this task simply and roughly. Either they need to train a model with large number of standardized dataset, or they ignore the domain knowledge and expertise of TCM. In order to solve these problems, we propose an hybrid neural network architecture for TCM prescription generation—PreGenerator. It includes a novel hierarchical retrieval mechanism, which can automatically extract prescription and herbal templates to facilitate accurate clinical prescription generation. Firstly, PreGenerator uses the Symptom-Prescription Retrieval (SHR) module to retrieve the most relevant prescriptions for a given patient's symptoms. In order to follow the rule of compatibility of herbs, the Herb-Herb Retrieval (HHR) module is introduced to retrieve the next most relevant herb according to the conditioned generated herbs. Finally, the prescription decoder (PreD) fuses the symptom features, the retrieved prescription and herbal template features to generate the most relevant and effective Chinese medicine prescription. The validity of the model is verified by automatic evaluation and manual evaluation on the real medical case dataset. In addition, our model can recommend herbs that do not appear on the prescription label but are useful for relieving symptoms, which shows that our model can learn some interactions between herbs and symptoms. This research also lays a foundation for the future research on intelligent query and prescription generation of traditional Chinese medicine.