Abstract: With the rapid growth of video surveillance data, there is an increasing demand for video anomaly detection, and reconstruction error detection methods based on deep autoencoders have been widely discussed. However, the autoencoder generalizes well, can reconstruct the anomaly well and lead to missed detection. In order to solve this problem, this paper proposes to adopt a memory module to enhance the autoencoder, which is called the Memory-augmented autoencoder (Memory AE) method. Given the input, Memory AE first obtains the encoding from the encoder, and then uses it as a query to retrieve the most relevant memory items for reconstruction. In the training phase, the memory content is updated and encouraged to represent prototype elements of normal data. In the test phase, the learned memory elements are fixed, and reconstruction is obtained from several selected memory records of normal data, thus the reconstruction will tend to be close to normal samples. Therefore, the reconstruction of abnormal errors will be strengthened for abnormal detection. Experiments on two public video anomaly detection datasets, namely Avenue dataset and ShanghaiTech dataset, proves the effectiveness of the proposed method.